Drop Python 3.9, use 3.10 for dist, bump Ledger by Sjors · Pull Request #819 · bitcoin-core/HWI

Sjors · 2026-01-21T15:54:56Z

This PR first switches the deterministic builds to Python 3.10. It then drops the end-of-life 3.9.

Then it bumps Speculos and Ledger and enables the tests that were previously disabled.

There's also a few minor cleanup commits, plus changes to make the Docker containers Podman friendly so I can easily run them locally.

Sjors · 2026-01-30T10:03:03Z

Rebased after #818 landed, ready for review.

I lightly tested the dist builds on Ubuntu (x86_64, including qt) and macOS (only command line).

I ran the GUI in Wine, but that doesn't have USB access so not very informative. I also ran it on a native Windows machine. Windows Defender wasn't happy about it, but I didn't do any code signing. There it detected a connected device, so that's good.

I wonder if it makes sense for the deterministic binary releases to jump straight ahead to a more modern Python?

Also, would it make sense to do one final HWI release with Python 3.9 before merging this?

Sjors · 2026-01-30T13:24:09Z

Added commit to re-enable the Ledger tests.

Sjors · 2026-01-31T09:04:48Z

Rebased just in case after recent merges.

achow101 · 2026-02-03T06:40:11Z

Does the dist build work on in an arm docker container, as described in the release process?

Sjors · 2026-02-03T11:20:59Z

Does the dist build work on in an arm docker container

I built inside an arm64 docker container on the x86 linux machine. I also made it do an hwi --help.

Rebased again, this time using generate_setup.py in the relevant commits, so e.g. the description field is updated when I touch the README.

The first time I ran that command, it moved hwilib.ui from packages to ui/* in package_data, see 35aa66a. No idea if that's correct.

I also built and tested the x86 macOS build on a native machine. I was able to run the QT application and it connected to a test deviced and fetch the descriptors.

Checksums:

19ee9fa6181421de1d6b404145438a1d00c61b3f0e55cc19a344f54fb5e76289  hwi-3.2.0-rc.1-linux-aarch64.tar.gz
3987142f1a441f893c28372b7ccd6a143b06c324cc9ee50f86b9cdb4d21f3204  hwi-3.2.0-rc.1-linux-aarch64.tar.gz/hwi
9909786e92a9cf0079d4aba986a6f8b856abba49c52a6df9e57dcae4c2e3c3a7  hwi-3.2.0-rc.1-linux-x86_64.tar.gz
64b5568167acd0d619f34c1739c430491b214bbf326bd8a4a2e0528c6b58d040  hwi-3.2.0-rc.1-linux-x86_64.tar.gz/hwi
97554105f474ba60b46d3b26534d254318114c05f5193c53e79bdf7167836239  hwi-3.2.0-rc.1-linux-x86_64.tar.gz/hwi-qt
83ac0f79ec5d9ecfb6aca98d325aaf05e47b3ba31b5c1e4fa69a2bfb088719d4  hwi-3.2.0-rc.1-mac-arm64.tar.gz
f9e884b02c2cd37e43f0668d32afef43cda649262d8ec891302a6d63f95026ed  hwi-3.2.0-rc.1-mac-arm64.tar.gz/hwi
6c070537a9f547c5b1e63a0c6d42aff9658a4e9305fa501c9e4ee752b04cdf3f  hwi-3.2.0-rc.1-mac-x86_64.tar.gz
241dea62a11ae2e18a547d8dd496e30cff7e005256ef460b6fde77aa303c6f40  hwi-3.2.0-rc.1-mac-x86_64.tar.gz/hwi
eea44149abb0a800c18f3895ea7b4f92937d01378bbfaa68ae74f89b4ef37aa3  hwi-3.2.0-rc.1-mac-x86_64.tar.gz/hwi-qt
be2d6ddb0fc18be6d9525c5004bc899e487e20ca51bd20aaa3e8e3ff43b28d0a  hwi-3.2.0-rc.1-windows-x86_64.zip
bf9e8a41e84537e1e3fa2310e8453a13cce50ee5befd8d6767e546bc29c17e2b  hwi-3.2.0-rc.1-windows-x86_64.zip/hwi-qt.exe
04b63d2a0b1e73253fe167aa7d9fe10e4da05f6c416319846b927deb00b000b1  hwi-3.2.0-rc.1-windows-x86_64.zip/hwi.exe
902438e1e1e988ab99099895f515ef701fc813fa42ad1620a733a6745f25933c  hwi-3.2.0rc1-py3-none-any.whl
f01ac678aa1c4da6e84eb83d42044a559acd0177313dd77b49a9494f9db9b7d6  hwi-3.2.0rc1.tar.gz

Sjors · 2026-02-03T14:12:45Z

The spurious Ledger failure appears to be the result of the test suite occasionally hitting the Cancel / Reject button.

2026-02-03T12:43:44.9074661Z seproxyhal: received (tag: 0x60, size: 0x02): b'\x00\x00'
2026-02-03T12:43:44.9074759Z automation: getting actions for "" (0, 0)
2026-02-03T12:43:44.9074882Z automation: getting actions for "Reject" (46, 36)
2026-02-03T12:43:44.9075003Z seproxyhal: applying automation ['button', 1, True]
2026-02-03T12:43:44.9075128Z seproxyhal: applying automation ['button', 1, False]
2026-02-03T12:43:44.9075220Z seproxyhal.packet: send 05000100
...
2026-02-03T12:43:44.9077945Z apdu: < 6985

Where 0x6985: DenyError according to device_exception.py

Added a commit to drop that rule, since it's unused anyway.

Sjors · 2026-07-03T15:29:40Z

Rebased just in case.

bumped the Python patch version to 3.10.20.
bumped Speculos to v0.26.9
bumped ledger-app-builder to 5.3.2
bumped Ledger app to 2.4.6
- plus two fixes from the develop branch
- some test automation changes
- the new version restricts xpub derivations, which test: make device signing cases more granular compensates for. This only impacts getkeypool.

I briefly tested the x86 linux gui.

Sjors · 2026-07-03T15:39:14Z

Added a commit here to fix Trezor CI. Unrelated so can be its own PR if it works.

Sjors · 2026-07-03T16:47:22Z

The one bitbox01 and one ledger failures seem spurious. But the jade test failures were real, so I'm pushing a new fix shortly (basically #832).

Sjors · 2026-07-03T19:40:38Z

Actually #832 did not do the trick. This time I ran the simulator locally to reproduce the issue. It was the PSBTv2 change on Bitcoin Core's master (again, this change bit me multiple times). I dropped the cbor commits, because they're not needed here, but based on local testing they should not be a problem.

I opened a fresh PR for just the CI fixes: #836

andreasgriffin

LGTM and I let codex also review with no issues found.

Sjors · 2026-07-04T09:28:12Z

The one failing ledger just is spurious.

@andreasgriffin much of this was written with the help of Claude and Codex, so I wouldn't expect it to find anything. Unless you give it a more specific prompt based on your own review, of something that might be incorrect.

In general for this Python bump it would be great if someone can test the binaries on every supported platform. I did that before the rebase, so I don't expect anything new to have broken, but still.

Sjors · 2026-07-04T10:33:43Z

Fable discovered that the one flaky failure here was due to a base58 address ending in To, which ended up at the start of the screen when the address was split. This collides with the automation rule "To".

It's a pre-existing bug, but I kept the fix be87c6e here, because it only impacts tests that were disabled before this PR.

Sjors · 2026-08-01T18:24:46Z

@achow101 Codex says it's your fault :-)

I can reproduce a successful no-cache ARM64 build both natively [on macOS] and under QEMU on x86_64 [Ubuntu]. The resulting image contains runnable aarch64 Python 3.10.20. The zero-valued configure probes suggest a local QEMU/binfmt issue; can you check docker buildx inspect --bootstrap and that /proc/sys/fs/binfmt_misc/qemu-aarch64 includes the F flag?

Sjors · 2026-08-01T19:19:25Z

c0990a1 is incorrect and should be dropped.

Done, but added a184337 build: generate UI before building distributions, because it seems the original commit stemmed from confusion around the sequence of build scripts.

achow101 · 2026-08-01T20:54:12Z

Codex says it's your fault :-)

Well, since I do the releases, if it doesn't work on my machine, it ain't gonna be merged.

Sjors · 2026-08-02T09:40:47Z

@achow101 did the suggestion work? I can't reproduce the issue, so will need more details.

achow101 · 2026-08-04T02:36:11Z

The current error is

 > [6/6] RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20':                
4.843 Downloading Python-3.10.20.tar.xz...                                                                                                                                    
4.844 -> https://www.python.org/ftp/python/3.10.20/Python-3.10.20.tar.xz                                                                                                      
11.22 Installing Python-3.10.20...                                                                                                                                            
11.33 patching file Makefile.pre.in
11.33 Hunk #1 succeeded at 787 (offset 46 lines).
438.4 
438.4 BUILD FAILED (Debian GNU/Linux 12 using python-build 2.8.1)
438.5 
438.5 Inspect or clean up the working tree at /root/.pyenv/sources/3.10.20
438.5 Results logged to /tmp/python-build.20260804021345.430.log
438.5 
438.5 Last 10 log lines:
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/codecs.o Python/codecs.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/compile.o Python/compile.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/context.o Python/context.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/dynamic_annotations.o Python/dynamic_annotations.c
438.6 make: *** [Makefile:1858: Objects/setobject.o] Segmentation fault (core dumped)
438.6 make: *** Waiting for unfinished jobs....
438.6 Objects/obmalloc.c:1413:1: warning: 'always_inline' function might not be inlinable [-Wattributes]
438.6  1413 | arena_map_get(block *p, int create)
438.6       | ^~~~~~~~~~~~~
------
build.Dockerfile:39
--------------------
  37 |     ENV BUILD_DATE="Jan  1 2026"
  38 |     ENV BUILD_TIME="00:00:00"
  39 | >>> RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'
  40 |     
  41 |     ENV LC_ALL=C.UTF-8
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c /bin/bash -c 'eval \"$(pyenv init --path)\" && eval \"$(pyenv virtualenv-init -)\" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'" did not complete successfully: exit code: 1

achow101 · 2026-08-04T04:51:30Z

Apparently I have to disable ASLR...

achow101 · 2026-08-04T04:52:21Z

+          git clone --branch 2.4.6 --depth 1 https://github.com/LedgerHQ/app-bitcoin.git
          cd app-bitcoin
-          make DEBUG=1 BOLOS_SDK=$NANOX_SDK
+          # Work around register-wallet policy parsing/display bugs fixed after v2.4.6.


In 50da3f1 "Bump Speculos and Ledger Bitcoin app"

Can we instead pin a commit that has these included instead of cherry picking them? We're already building from source anyways.

I bumped to 2.5.0, which has been released since and contains these fixes.

achow101 · 2026-08-04T04:53:29Z

+    # Work around -Werror build failures in Speculos' bundled deps.
+    # GCC < 15 errors out on unknown "-Wno-error=..." options, so only add the
+    # unterminated-string-initialization suppression when the compiler supports it.
+    CFLAGS="-O -fno-builtin -fPIC -Wall -Wextra -Werror -Wno-error=maybe-uninitialized -Wno-error=array-parameter -Wno-error=array-bounds -Wno-error=discarded-qualifiers"


In 50da3f1 "Bump Speculos and Ledger Bitcoin app"

Where did these flags come from? I don't see them in the speculos codebase at all.

Dropped. Not sure if this was useful in earlier versions (it's been there since at least January), but it's definitely not needed now.

It looks like my Ubuntu test machine needs it, but CI doesn't, so I will leave it out.

Most device simulators accept arbitrary keypool paths, so keep testing that behavior by default. Some devices enforce their own derivation path policies. Let those emulators opt out of the arbitrary-path portion while still running the remaining keypool checks.

Sjors · 2026-08-04T08:49:22Z

Rebased after #838 and addressed inline feedback.

Sjors · 2026-08-04T16:25:07Z

The Ledger app bump from 2.4.6 to 2.5.0 required another automation update, due to LedgerHQ/app-bitcoin@d452557, LedgerHQ/app-bitcoin@26f631c and LedgerHQ/app-bitcoin@f883a91.

Ledger app 2.5.0 can take over 60 seconds to sign the large transaction fixture. Allow stdin commands up to 120 seconds.

It's unused and occasionally trips up a test.

Limit the warning rule to the centered title so it does not also click the repeated header on the details screen. This prevents the queued extra click from selecting Back to safety before Continue anyway is confirmed.

Sjors · 2026-08-04T18:44:32Z

I added cd8a431 to double the timeout for the ledger stdin jobs. IIUC the bindist and cli jobs quickly fail the huge psbt test, whereas stdin actually processes it and that takes too long.

Part of the problem is that logging is so verbose, that it slows the test down.

@bigspider can you demote the packet/byte chatter from DEBUG to TRACE? The (more interesting) screen read and button press information is also at DEBUG, so it's currently non-trivial to disentangle. And log files are ~400MB.

achow101 · 2026-08-04T19:31:00Z

can you demote the packet/byte chatter from DEBUG to TRACE? The (more interesting) screen read and button press information is also at DEBUG, so it's currently non-trivial to disentangle. And log files are ~400MB.

I think dropping --log-level seproxyhal:DEBUG from our emulator command would drop those messages.

Sjors · 2026-08-04T19:43:55Z

@achow101 it does, but we also lose the (very useful) seproxyhal: applying automation [...] messages.

Sjors force-pushed the 2026/01/python-bump branch 2 times, most recently from 8489c74 to 57f6690 Compare January 23, 2026 09:21

Sjors force-pushed the 2026/01/python-bump branch from 57f6690 to a608ea3 Compare January 30, 2026 10:00

Sjors marked this pull request as ready for review January 30, 2026 10:01

Sjors mentioned this pull request Jan 30, 2026

test: diagnose LedgerX test failures #809

Closed

Sjors force-pushed the 2026/01/python-bump branch from a4e3a84 to ae9e49d Compare January 31, 2026 09:04

Sjors force-pushed the 2026/01/python-bump branch from ae9e49d to f473a2e Compare February 3, 2026 11:20

Sjors mentioned this pull request Feb 11, 2026

GUI: Migrate to PySide6 (Enables ARM/Apple Silicon support) #824

Open

Sjors mentioned this pull request Jul 3, 2026

build: use Python 3.10 #790

Closed

Sjors force-pushed the 2026/01/python-bump branch from 3ac16e0 to 5197ac2 Compare July 3, 2026 15:29

Sjors force-pushed the 2026/01/python-bump branch from c85d39c to a8bd342 Compare July 3, 2026 15:57

Sjors force-pushed the 2026/01/python-bump branch from d2093d9 to d21b545 Compare July 3, 2026 17:03

Sjors mentioned this pull request Jul 3, 2026

Fix CI ... again #836

Merged

Sjors force-pushed the 2026/01/python-bump branch from d21b545 to 712823a Compare July 3, 2026 19:48

This was referenced Jul 3, 2026

MuSig2 #794

Draft

Extend to python 3.13 and python 3.14 #835

Open

andreasgriffin reviewed Jul 4, 2026

View reviewed changes

Sjors force-pushed the 2026/01/python-bump branch from 712823a to 887843c Compare July 4, 2026 10:33

Sjors force-pushed the 2026/01/python-bump branch from 831193d to 76bd097 Compare August 1, 2026 19:18

achow101 reviewed Aug 4, 2026

View reviewed changes

achow101 mentioned this pull request Aug 4, 2026

Bump dependencies and modernize pyproject.toml #843

Draft

Sjors force-pushed the 2026/01/python-bump branch from 76bd097 to 8ae811e Compare August 4, 2026 08:35

Sjors added 6 commits August 4, 2026 10:36

build: generate UI before building distributions

8603233

build: add docker / container ignore files

68c51cf

build: use Podman friendly syntax

5c9d86d

build: deterministic builds use Python 3.10

45de0be

Drop Python 3.9 support

edfeace

Sjors force-pushed the 2026/01/python-bump branch from 8ae811e to ac68456 Compare August 4, 2026 08:49

Sjors force-pushed the 2026/01/python-bump branch from ac68456 to 0bf6d7a Compare August 4, 2026 16:25

Sjors added 6 commits August 4, 2026 20:31

Bump Speculos and Ledger Bitcoin app

976353e

test: increase stdin command timeout

cd8a431

Ledger app 2.5.0 can take over 60 seconds to sign the large transaction fixture. Allow stdin commands up to 120 seconds.

test: reenable LedgerX tests

d2a5874

Skip archiving hwi-qt on non-x86 platforms

1b0cbea

Drop Ledger deny rule from tests

0379e27

It's unused and occasionally trips up a test.

test: disambiguate Ledger warning automation

d27d196

Limit the warning rule to the centered title so it does not also click the repeated header on the details screen. This prevents the queued extra click from selecting Back to safety before Continue anyway is confirmed.

Sjors force-pushed the 2026/01/python-bump branch from 0bf6d7a to d27d196 Compare August 4, 2026 18:44

Uh oh!

Conversation

Sjors commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Jan 30, 2026

Uh oh!

Sjors commented Jan 31, 2026

Uh oh!

achow101 commented Feb 3, 2026

Uh oh!

Sjors commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Feb 3, 2026

Uh oh!

Sjors commented Jul 3, 2026

Uh oh!

Sjors commented Jul 3, 2026

Uh oh!

Sjors commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Jul 3, 2026

Uh oh!

andreasgriffin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sjors commented Jul 4, 2026

Uh oh!

Sjors commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

achow101 commented Aug 1, 2026

Uh oh!

Sjors commented Aug 2, 2026

Uh oh!

achow101 commented Aug 4, 2026

Uh oh!

achow101 commented Aug 4, 2026

Uh oh!

achow101 Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Sjors Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

achow101 Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Sjors Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Sjors Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Sjors commented Aug 4, 2026

Uh oh!

Sjors commented Aug 4, 2026

Uh oh!

Sjors commented Aug 4, 2026

Uh oh!

achow101 commented Aug 4, 2026

Uh oh!

Sjors commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Sjors commented Jan 21, 2026 •

edited

Loading

Sjors commented Jan 30, 2026 •

edited

Loading

Sjors commented Feb 3, 2026 •

edited

Loading

Sjors commented Jul 3, 2026 •

edited

Loading

andreasgriffin left a comment •

edited

Loading

Sjors commented Jul 4, 2026 •

edited

Loading

Sjors commented Aug 1, 2026 •

edited

Loading

Sjors commented Aug 1, 2026 •

edited

Loading